About objective dialogue evaluation methods

'Objective' evaluation means evaluation using numbers (metrics) which can be calculated without intervention by humans. An obvious advantage is that human eeort is reduced. It is also claimed that it is less biased by human opinion, but in this text it is shown that there are a lot of choices to be made when applying these methods, which introduces bias at a diierent level. Two objective evaluation methods which are claimed to be state-of-the-art, both developed in AT&T, are described and commented upon. Both are black-box methods, meaning that they can be used without referring to the implementation of the dialogue system.